Super-resolution device and method

ABSTRACT

A super-resolution device and method for setting at least one of a plurality of pixels included in image data as target pixels, the image data including pixels arranged in a screen and pixel values representing brightness, an area including the target pixel and peripheral pixels as a target area, and an area for searching pixel value change patterns in the target pixel area; calculating a difference between a first change pattern and second change pattern; comparing a difference between the first and second change patterns; calculating a pixel value of a super-resolution image having a number of pixels larger than a number of pixels included in the image data on the basis of a decimal-accuracy-vector, an extrapolated vector, and pixel values obtained from the image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a division of and claims the benefit of priorityunder 35 U.S.C. §120 from U.S. Ser. No. 11/828,397 filed Jul. 26, 2007,and claims the benefit of priority under 35 U.S.C. §119 from JapanesePatent Application No. 2006-276128 filed Oct. 10, 2006, the entirecontents of each of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a super-resolution device and method.

2. Description of the Related Art

TVs or displays having a large number of pixels and high resolution arenow in widespread use. These TVs or the displays convert a number ofpixels in the image data into a number of pixels of a panel whendisplaying an image. In the conversion of super-resolution forincreasing the number of pixels, a multiple frame deterioration reverseconversion method is conventionally used for obtaining an image sharperthan what is possible with a conventional linear interpolation method(for example, see U.S. Pat. No. 6,285,804, S. Park, et al. “Superresolution Image Reconstruction: A Technical Overview,” IEEE SignalProcessing Magazine, USA, IEEE May 2003, p. 21-36), the contents ofwhich are incorporated herein by reference).

Taking advantage of the fact that the photographic subject which comesout in a reference frame also comes out on another frame, the multipleframe deterioration reverse conversion method detects the movement ofthe photographic subject with a high degree of accuracy at a pixelinterval or lower and obtains a plurality of sample values in which theposition is minutely shifted with respect to an identical local positionof a photographic subject.

In the multiple frame deterioration reverse conversion method, a numberof low-resolution images are necessary to obtain a sufficient number ofsample values, and hence the amount of memory increases. There is also aproblem that it is necessary to obtain the relation of a number ofcorresponding points by a search process of block matching, and hencethe amount of computation increases.

SUMMARY OF THE INVENTION

In view of such circumstances, it is an object of the invention toprovide a super-resolution device and method for obtaining a sharpsuper-resolution image with small amount of memory and computation.

In order to solve the above-described object, an aspect of the inventionis a super-resolution device including:

a candidate area setting unit that sets at least one of a plurality ofpixels included in an image data as a target pixel, the image dataincluding the plurality of pixels arranged in a screen and pixel valuesrepresenting the brightness of the pixels, sets an area including thetarget pixel and pixels in the periphery of the target pixel as targetpixel area, and sets a search area for searching a plurality of changepatterns of the pixel values of the pixels included in the target pixelarea within the screen;

a matching difference calculating unit that calculates differencesbetween the change pattern of the pixel values of the pixels included inthe target pixel area and the change pattern of the pixel values of thepixels included in the area, the pixels in the area including thesearched pixel in the search area and the pixels in the periphery of thesearched pixels;

a difference comparing unit that compares differences of the changepattern of the respective pixels in the search area calculated by thematching difference calculating unit to obtain a first pixel positionwith the minimum difference and a second pixel position in the peripheryof the first pixel position with a second difference thereof;

a memory that stores the first pixel position and a first differencethereof, the second pixel position and a second difference thereofcalculated by the difference comparing unit;

a decimal-accuracy-vector calculating unit that calculates a positionwith the minimum difference in the search area with a decimal accuracyon the basis of the first pixel position and the first differencethereof and the second pixel position and the second difference thereofstored in the memory, and calculates a decimal-accuracy-vector startingfrom the target pixel and terminating at the position with the minimumdifference;

an extrapolated vector calculating unit that calculates an extrapolatedvector of the decimal-accuracy-vector terminating at the pixel on thescreen which is not included in the search area using thedecimal-accuracy-vector; and

a super-resolution pixel value calculating unit that calculates a pixelvalue of a super-resolution image having the number of pixels largerthan the number of pixels included in the image data on the basis of thedecimal-accuracy-vector, the extrapolated vector, and the pixel valuesobtained from the image data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a super-resolution device according to anembodiment of the invention;

FIG. 2 is a block diagram of a super-resolution device according to theembodiment of the invention;

FIG. 3 is a flowchart showing an example of the operation of thesuper-resolution device according to the embodiment of the invention;

FIG. 4 is a drawing showing a positional relation between a screen andpixels of a low-resolution image data;

FIG. 5 is a drawing showing a super-resolution image obtained bysuper-resolution on the image shown in FIG. 4;

FIG. 6 is a drawing showing a low-resolution image obtained by matchingthe pixel interval of the image in FIG. 4 with the pixel interval of theimage in FIG. 5;

FIG. 7 is a drawing showing a positional relation between the pixels inFIG. 4 and in FIG. 5;

FIG. 8 is a drawing showing a relation between the positional coordinateand the brightness of a photographic data;

FIG. 9 is a drawing showing a setting of the target pixel and the targetimage area;

FIG. 10 is a drawing showing the setting of the target pixel and thesearch area;

FIG. 11 is a drawing for explaining the parabola fitting method;

FIG. 12 is a drawing showing calculation of a self-congruent position bya matching processing;

FIG. 13 is a drawing showing generation of the self-congruent positionby estimation by extrapolation;

FIG. 14 is a drawing showing generation of the self-congruent positionby estimation by interpolation;

FIG. 15 is a drawing showing generation of the self-congruent positionby duplication;

FIG. 16 is a drawing showing a plurality of self-congruent positionscalculated in the screen space;

FIG. 17 is a flowchart showing an example of the operation when thepixel value of the super-resolution image is obtained by thesuperimposing method;

FIG. 18 is a drawing showing a screen, pixels, and squares forexplaining a method of calculating a sample value of an initiallyestimated super-resolution image;

FIG. 19 is a flowchart showing an example of the operation forsuper-resolution by establishing conditional expressions for each samplevalue; and

FIG. 20 is a drawing showing a relation between the positionalcoordinate and the brightness after the super-resolution.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings, a super-resolution device and methodaccording to embodiments of the invention will be described.

The invention is not limited to the embodiments shown below, and may beimplemented by selecting or modifying in various manner.

FIG. 1 is a block diagram of a super-resolution device according to anembodiment of the invention.

As shown in FIG. 1, the super-resolution device includes a memory 101, acandidate area setting unit 102, a matching difference calculating unit103, a difference comparing unit 104, a memory 105, a super-resolutionpixel value calculating unit 106, a parabola fitting unit 107, a memory108, and a self-congruent position estimating unit 109. In thisspecification, the term “self-congruent” means that the brightnesschanging pattern of the pixels are similar in the same frame. The term“self-congruent position” is the position of the self-congruentexpressed by vector.

The memory 101 acquires a low-resolution image data and stores the same.The low-resolution image data may be a movie or a still image, and is animage data obtained by arranging a plurality of pixels in a screen andexpressing the brightness of the pixels in pixel values. In thisembodiment, the low-resolution image data is acquired from an imagesource, that is, from an image data generating unit (not shown) such asa camera or a TV. More specifically, the low-resolution image data is animage data taken by a camera or an image data received by the TV.

The candidate area setting unit 102 determines at least one of theplurality of pixels of the low-resolution image data as a target pixeland an area including the target pixel and pixels in the periphery ofthe target pixel as a target pixel area, and sets a search area forsearching a plurality of change patterns of the pixel values of thepixels included in the target pixel area in the screen.

Then, the candidate area setting unit 102 generates signals whichindicate the target pixel, the target area, and the search area, andoutputs these signals to the memory 101 and the memory 105.

Based on the signals which indicate the target pixel, the target area,and the search area, the memory 101 outputs an image data of the targetpixel area including the target pixels and the image data in the searcharea from the low-resolution pixel image data to the matching differencecalculating unit 103. The memory 101 supplies a low-resolution imagedata to the super-resolution pixel value calculating unit 106 one byone.

The matching difference calculating unit 103 calculates a differencebetween a change pattern of the pixel values of the pixels included inthe target pixel area and the change pattern of the pixel values of thepixels included in the search area, the pixels in the area including thesearched pixels in the search area and the pixels in the periphery ofthe searched pixels.

The matching difference calculating unit 103 calculates a differencebetween the image data within the target pixel area and the image datawithin the search area. The difference is calculated, for example, bysum of absolute distance or sum of square distance of the respectivepixel values. The image data of the target pixel area may be, forexample, data of a target block. The matching difference calculatingunit 103 changes the image portion in the search area whose differenceis to be calculated in sequence and obtains a difference with respect tothe image data of the image portion in the target pixel area whosedifference is to be calculated.

The difference comparing unit 104 calculates the position of a pixelwhich has the smallest difference out of the plurality of differences inthe search area calculated by the matching difference calculating unit103.

The memory 105 acquires positional information from the candidate areasetting unit 102, and stores the position of the pixel having thesmallest difference calculated by the difference comparing unit 104 andthe matching difference, and the positions of pixels around the positionof the pixel having the smallest difference and the matching differenceat these positions.

The parabola fitting unit 107 applies symmetric function on the basis ofthe position of the pixel having the smallest difference and thematching difference, and the positions of the pixels around the positionof the pixel having the smallest difference and the matching differenceat these positions store in the memory 105, calculates a position havingthe smallest matching difference with a decimal accuracy, and determinesthe calculated position as a self-congruent position. At least oneself-congruent position is obtained for one target pixel. Detaileddescription of the parabola fitting unit 107 will be given later.

The self-congruent position estimating unit 109 estimates and calculatesat least one self-congruent position on the basis of the amount ofchange of the self-congruent position calculated by the parabola fittingunit 107.

The memory 108 stores information on the self-congruent positionobtained by the parabola fitting unit 107 and the self-congruentposition estimating unit 109.

After having obtained the self-congruent position for the predeterminedpixel of the low-resolution image, the super-resolution pixel valuecalculating unit 106 obtains image data of the low-resolution image fromthe memory 101 and obtains the self-congruent position from the memory108, establishes conditional expressions simultaneously using theself-congruent position for each pixel data of the low-resolution image,obtains a solution to determine the pixel value of the super-resolutionimage, and outputs the pixel value data.

Subsequently, referring to FIG. 2, the super-resolution device in a casein which an over sampling method is used instead of the parabola fittingmethod will be described.

The super-resolution device in FIG. 2 is configured in such a mannerthat the parabola fitting unit 107 is removed from the configuration inFIG. 1, and an over sampling unit 110 and a memory 111 are addedinstead.

The candidate area setting unit 102 sets at least one of the pluralityof pixels included in the image data as the target pixel, the image dataincluding the plurality of pixels arranged in a screen and pixel valuesrepresenting the brightness of the pixels, sets an area including thetarget pixel and the pixels in the periphery of the target pixel as thetarget pixel area, and sets a search area for searching a plurality ofpatterns of change of the pixel values of the pixels included in thetarget pixel area.

The over-sampling unit 110 interpolates another pixel between the pixelsof the image data whose target pixel area and the search area are set togenerate an interpolated image data. In other words, the over-samplingunit 110 increases the data amount of the low-resolution data bydepending on the intervals of difference calculation.

The memory 111 stores data sampled by the over-sampling unit 110temporarily and supplies the data to the matching difference calculatingunit 103.

The matching difference calculating unit 103 calculates a differencebetween the change pattern of the pixel values of the pixels included inthe target pixel area and the change pattern of the pixel values of thepixels included in an area including the searched pixel in the searcharea and the pixels in the periphery of the searched pixel.

The difference comparing unit 104 calculates the pixel position havingthe smallest difference out of the plurality of differences in thesearch area calculated by the matching difference calculating unit 103.

The memory 105 acquires positional information about the calculatedpixel position having the smallest matching difference calculated by thedifference comparing unit 104 from the candidate area setting unit 102,and stores the integral-accuracy-vector starting from the target pixeland terminating at the pixel having the smallest matching difference.

The self-congruent position estimating unit 109 estimates and calculatesone or more self-congruent positions on the basis of the differencecalculated in the matching difference calculating unit 103 and thechange amount of the integral accuracy vector calculated by the memory105.

The memory 108 stores information of the self-congruent positionobtained by the self-congruent position estimating unit 109.

After having obtained the self-congruent position of the predeterminedpixels of the low-resolution image, the super-resolution pixel valuecalculating unit 106 obtains the pixel data of the low resolution imagefrom the memory 101 and the self-congruent position from the memory 108,establishes conditional expressions simultaneously using theself-congruent position for each pixel data of the low-resolution image,obtains a solution to determine the pixel value of the super-resolutionimage, and outputs the pixel value data.

Referring now to FIG. 3, an embodiment of the operation ofsuper-resolution device described in conjunction with FIG. 1 will bedescribed. The image may be referred to as a frame in the followingdescription.

As shown in FIG. 3, in Step S201, the candidate area setting unit 102sets a pixel of the low-resolution image data as a target pixel in apredetermined sequence. The sequence is, in the case of a still image, araster sequence, for example, rightward from the upper left pixel in thescreen, downward from the upper row.

Subsequently, in Step S202, the matching difference calculating unit103, the difference comparing unit 104 and the parabola fitting unit 107detect a point corresponding to the target pixel (self-congruentposition) in a screen space of the low-resolution image data.

Subsequently, in Step S203, the self-congruent position estimating unit109 estimates and generates a new self-congruent position on the basisof the change amount of the self-congruent position calculated by theparabola fitting unit 107.

Subsequently, in Step S204, the matching difference calculating unit 103determines whether or not the self-congruent position is obtained foreach pixel of the low-resolution image data used for super-resolution.If No, the procedure goes back to Step S201, in which the next pixel isprocessed, and if Yes, the procedure goes to Step S205.

Subsequently, in Step S205, the super-resolution pixel value calculatingunit 106 calculates a pixel value of the super-resolution image datacorresponding to the low-resolution image data using the pixel value ofthe low-resolution image data and the detected self-congruent positionand terminates the process. Calculation of the pixel value of thesuper-resolution image data will be described referring to FIG. 16.

FIG. 4 shows a positional relation between a screen 301 and a pixel 302of the low-resolution image.

The image basically has brightness which is continuously distributed inthe screen space. However, in the case of the digital image data handledhere, pixels are arranged in the screen space as discrete sample points,and the ambient brightness thereof is represented by the brightness ofeach pixel by itself.

FIG. 4 shows a state in which the screen is divided into twenty-foursquares arranged to have six in the lateral direction and four in thevertical direction, and twenty-four pixels 301 are arranged at thecenters thereof as the sample points 302.

Subsequently, a state in which the super-resolution is applied to thescreen shown in FIG. 4 by double in the lateral direction and double inthe vertical direction is shown. Sample points 401 of the pixels in thesuper-resolution image data are indicated by hollow circles. In thismanner, the interval of the sample points 401 of the pixels is half ofthe low-resolution image data.

FIG. 6 shows the pixels of the original low-resolution image data at theinterval which is the same as that of the super-resolution image data.In this case, the size of the low-resolution image data is smaller thanthat of the super-resolution image data.

In this manner, when the size of the screen of the low-resolution imagedata is adjusted to match the screen of the super-resolution image data,the interval of the sample points of the pixels increases, and when theinterval of the sample points of the pixels is adjusted to match that ofthe super-resolution image data, the size of the screen is reduced.However, these phenomena represent the same thing, and hence in thisspecification, the low-resolution image is shown as in FIG. 4 and FIG. 6as needed for the sake of convenience of description.

FIG. 7 is a drawing showing the sample points of the pixels in thelow-resolution image data with solid circles, and the sample points ofthe pixels in the super-resolution image data with hollow circles. Theprocess of the super-resolution is to obtain the brightness values ofthe sample points represented by the hollow circles on the basis of thebrightness values provided to the sample points represented by the solidcircles.

Subsequently, using FIG. 8 to FIG. 11, Step S202 described in FIG. 3will be described with a detailed example.

In the plurality of frames deterioration reverse conversion method inthe related art, the super-resolution is performed by increasing thenumber of sample points in the low-resolution image data by calculatingthe corresponding identical points among the multiple frames with subpixel accuracy. In other words, a large number of pixel values obtainedby sampling the portions having the same brightness change withdifferent phases are necessary among the multiple frames, and hence alarge amount of memory is necessary.

FIG. 8 shows data of an actual picture.

The lateral axis represents the lateral coordinate of the pixel and thevertical axis represents the brightness. Five rows of data arerepresented by different curved lines respectively. As will be seen,there are portions which demonstrate a very similar brightness changeeven though the row is different in the same frame. Such a property ofthe image is referred to as having a self-congruent property in thelocal pattern, and the self-congruent position existing around a certaintarget pixel is referred to as a self-congruent position.

In the invention, since the super-resolution is achieved using theself-congruent property of the photographic subject within the frame, itis not necessary to hold a plurality of low-resolution image data in thememory, and hence the super-resolution is achieved with a small amountof memory.

FIG. 9 is a conceptual drawing showing a state in which the candidatearea setting unit 102 described in FIG. 1 and FIG. 2 sets the targetpixel and the target area.

As shown in FIG. 9, the candidate area setting unit 102 takes outseveral square pixels, for example, a square block 803 having 5×5 pixelsor 3×3 pixels from a frame 802 with a target pixel 801 placed at thecenter.

FIG. 10 is a conceptual drawing showing a state in which the candidatearea setting unit 102 described in FIG. 1 and FIG. 2 sets a search area901. In this drawing, an example in which the search area is set withsix pixels in the x-direction with the y-coordinate fixed. The matchingdifference calculating unit 103 searches a portion whose change patternof the pixel value is close to the target image area 803 shown in FIG. 9for the respective pixels included in the search area 901.

The matching difference among the respective image areas to becalculated by the matching difference calculating unit 103 may be SSD(Sum of Square Distance) which is a sum of square distance among therespective pixel values in the image area or SAD (Sum of AbsoluteDistance) which is a sum of the absolute distance.

In this case, the search area is set in the x-direction with they-coordinate fixed. The method of obtaining the sub pixel estimation inthis manner is specifically effective when the brightness of thelow-resolution image data changes in the lateral direction.

Although not shown in the drawings, a method of fixing the x-coordinate,setting the search area to the y-direction and obtaining the sub pixelestimation is effective when the brightness of the low-resolution imagedata changes in the vertical direction.

Therefore, a method of setting at least one search area in the lateraldirection which is orthogonal to the direction of the edge if it isvertical, and at least one search area in the vertical direction if itis lateral by the candidate area setting unit 102 is effective. In otherwords, the direction of inclination of the pixel value of the targetpixel may be detected to search the self-congruent position in thedirection of inclination.

The positional information of the pixels from the candidate area settingunit 102 is called, the matching differences calculated by the matchingdifference calculating unit 103 are compared to obtain a pixel positionwith the minimum difference, and the position of the pixel with minimumdifference and the matching difference, and the positions of the pixelsin the periphery of the pixel with minimum difference and the matchingdifferences at these positions are stored in the memory.

Subsequently, estimation of the sub pixel (with a decimal accuracy) inthe preset search area will be described. One of the methods ofestimating the sub pixel is a parabola fitting method (for example, see“Signification and Property of Sub pixel Estimation in Image Matching”by Shimizu, Okutomi, the contents of which are incorporated herein byreference).

The parabola fitting method calculates a position with the minimummatching difference with a decimal accuracy from the matching differencebetween the target pixel area and the candidate image area around thepixel within the preset search area with an integral accuracy.

The matching difference is calculated by shifting the position of thecandidate image area in the search area with an integral accuracy, andthe matching difference map with an integral accuracy in the search areaspace is calculated.

FIG. 11 is a graph showing a matching method in the parabola fittingmethod, in which the lateral axis represents the pixel and the verticalaxis represents the matching difference.

As shown in FIG. 11, the positional shift amount at a sub pixel accuracycan be calculated as a position of an apex of a parabola (or a symmetriccontinuous function) applied to a discrete matching difference map withan integral accuracy by applying the parabola (or the symmetriccontinuous function) around the amount of positional shift (x=m) with anintegral accuracy with the smallest matching difference.

FIG. 12 is a drawing illustrating a state of calculating a vector with adecimal accuracy using the parabola fitting method.

As shown in FIG. 12, a position with the minimum difference iscalculated with a decimal accuracy in the search area 901 on the basisof the matching difference calculated at each pixel in the search area901, and a vector 1101 with a decimal accuracy starting from the targetpixel 801 and terminating at this position.

In addition to the parabola fitting method, an isometric fitting asdescribed in “Signification and Property of Sub pixel Estimation inImage Matching” by Shimizu, Okutomi may also be applied.

In the method using the over sampling unit 110 described in conjunctionwith FIG. 2, the low-resolution image data is enlarged to, for example,double by an enlargement method such as a linear interpolation or acubic convolution method. When the pixel accuracy is searched in thisstate, it is equivalent to the calculation of the shift amount with anaccuracy of 0.5 pixels in the original low-resolution image data. Inthis manner, in the over sampling method, it is necessary to double thedata to halve the accuracy (the interval to obtain the difference), andto quadruple the same to obtain a quarter accuracy.

Referring now to FIG. 13 to FIG. 15, generation of the self-congruentposition by estimation performed in Step S203 in FIG. 3 will bedescribed.

The self-congruent position calculation performed in Step S202 requiresa large amount of processing as it is necessary to execute thecalculation of the matching difference between the image areas in thesearch area by the number of times which corresponds to the number ofthe self-congruent positions to be obtained. Therefore, in the StepS203, new self-congruent positions are generated with a small amount ofprocessing by estimation by extrapolation, estimation by interpolationand estimation by duplication on the basis of the self-congruentpositions calculated in Step S202.

The estimation by extrapolation here means to estimate newself-congruent positions from the search area outside the one or moreself-congruent positions calculated by matching.

The estimation by interpolation means to estimate new self-congruentpositions from the search area positioned inside the two or moreself-congruent positions calculated by the matching.

Estimation by duplication means to estimate the self-congruent positionof the target pixel calculated by the matching as the self-congruentpositions of the target pixels nearby.

FIG. 13 is a drawing for explaining that the self-congruent positionsare generated by the estimation by extrapolation.

As shown in FIG. 13, a self-congruent position 1202 at one line above atarget pixel 1201 is calculated by the Step S202 in FIG. 3. Theself-congruent position at two lines above can be estimated to be thedouble the amount of change up to the self-congruent position 1202 atone line above the target pixel 1201 by extrapolation. Reference numeral1203 designates the self-congruent position at two lines above obtainedby estimation by extrapolation.

The estimation by extrapolation may be performed not only by estimatingone position from one self-congruent position, but performed byestimating plurality of self-congruent positions. It is also possible toestimate the new self-congruent position at a position at decimalmultiple in amount of change as well as the position at integralmultiple in amount of change.

In other words, by using the vector 1202 with a decimal accuracystarting from the target pixel and terminating at the position with theminimum difference calculated with a decimal accuracy in the search areaby the parabola fitting unit 107 in FIG. 1, or by using the vector 1202with an integral accuracy obtained by interpolating another pixelbetween the pixels of the image data in which the target pixel area andthe search area are set to generate an interpolated image data, and thencalculating the position with the minimum difference by the oversampling unit 110 in FIG. 2, the extrapolated vector 1203 with a decimalaccuracy terminating at a pixel on the screen which is not included inthe search area is calculated.

FIG. 14 is a drawing for explaining that a self-congruent position isgenerated by the estimation by interpolation.

As shown in FIG. 14, a self-congruent position 1302 at one line above atarget pixel 1301 and a self-congruent position 1303 at three linesabove thereof are calculated in Step S202 in FIG. 3. A self-congruentposition 1304 at two lines above can be estimated by interpolation as aninternally dividing point between the amount of change from the targetpixel position 1301 to the self-congruent position 1302 at one lineabove and the amount of change from the target pixel position 1301 tothe self-congruent position 1303 at three lines above.

The estimation by interpolation may estimate not only the singleself-congruent position 1304 from the two self-congruent positions 1302,1303, but also a plurality of self-congruent positions obtained bydividing internally into n equal parts.

In other words, by using the vector 1202 with a decimal accuracystarting from the target pixel and terminating at the position with theminimum difference calculated with a decimal accuracy in the search areaby the parabola fitting unit 107 in FIG. 1, or by using the vectors1302, 1303 with an integral accuracy obtained by interpolating anotherpixel between the pixels of the image data in which the target pixelarea and the search area are set to generate an interpolated image data,and then calculating the position with the minimum difference by theover sampling unit 110 in FIG. 2, the interpolated vector 1304 with adecimal accuracy terminating at a pixel on the screen which is notincluded in the search area is calculated.

FIG. 15 is a drawing for explaining that a self-congruent position isgenerated by the estimation by duplication.

As shown in FIG. 15, a self-congruent position 1402 at one line above atarget pixel 1401 is calculated by the Step S202 in FIG. 3. By copyingthe amount of change from the target pixel position 1401 to theself-congruent position 1402, the self-congruent position 1404 at oneline above the target pixel 1401 and a self-congruent position 1406 atone line below the target pixel 1401 can be generated.

In other words, by using the vector 1202 with a decimal accuracystarting from the target pixel and terminating at the position with theminimum difference calculated with a decimal accuracy in the search areaby the parabola fitting unit 107 in FIG. 1, or by using a vector 1402with an integral accuracy obtained by interpolating another pixelbetween the pixels of the image data in which the target pixel area andthe search area are set to generate the interpolated image data and thencalculating the position with the minimum difference by the oversampling unit 110 in FIG. 2, the congruent vector with the decimalaccuracy terminating at a pixel in the screen which is not included inthe search area is calculated.

As described above, by estimating the self-congruent position in StepS203 in FIG. 3, the self-congruent position can be calculated with thesmall amount of processing. In addition, the self-congruent positionscan be padded out, so that the image quality can be improved.Furthermore, since the self-congruent position at a position closer tothe target pixel such as to generate the self-congruent positions at 0.5line above and 1.5 line above the self-congruent position at one lineabove, whereby improvement of the image quality is achieved.

Referring now to FIG. 16, calculation of the pixel value of thesuper-resolution image data performed in Step S205 in FIG. 3 will bedescribed.

At the timing when the process in Step S204 in FIG. 3 is ended, forexample, the self-congruent positions as indicated by cross-signs inFIG. 16 are obtained. In this manner, although there are various mannersto obtain the values of pixels arranged in a lattice-like pattern fromsample points distributed non-uniformly, for example, when employing asuperimposing method (for example, non-uniform interpolation. See S.Park, et. al. “Super-Resolution Image Reconstruction: A TechnicalOverview” p. 25), the pixel value of the super-resolution image data canbe obtained by inspecting the sample values near there, and finding thesample value at the closest position to the pixel of thesuper-resolution image data and determining the sample value as thepixel value of the super-resolution image data. Alternatively, it isachieved by increasing the weight of the sample values as the distancefrom the pixel of the super-resolution image data is decreased, anddetermining the weighted average of the sample values as the pixel valueof the super-resolution image data. Further alternatively, an average ofsample values which are closer than a certain distance is employed as apixel value of the super-resolution image data.

Referring now to FIG. 17, a flowchart for obtaining the pixel value ofthe super-resolution image data by the superimposing method will bedescribed.

As shown in FIG. 17, in Step S1601, the distances to the respectivesample points are obtained for each pixel of the super-resolution imagedata.

Subsequently, in Step S1602, the respective pixel values are obtained asthe weighted average of the sample points. At this time, the closer thedistance of the sample values from the respective pixels, the more theweight is increased.

When POCS method (for example, see S. Park, et. al, “Super-ResolutionImage Reconstruction: A Technical Overview” p. 29) is used instead ofthe superimposing method, the process is more complicated, but a sharperimage can be obtained.

In the POCS method, an initially estimated super-resolution image isprovided to each pixel in the super-resolution image data by a bilinearinterpolating method or the cubic convolution method. Then, theestimated super-resolution images when the pixel values of the initiallyestimated super-resolution image of the super-resolution image data areused at the positions of the respective sample values are calculated.

Referring now to FIG. 18, a method of calculating a preliminary samplevalue will be described.

As shown in FIG. 18, a screen 1701 is divided into a plurality ofsquares 1702. The pixel values which represent the distribution of thebrightness of the respective squares are pixel values 1703 at thecenters thereof. The size of the square is determined by the density ofthe pixels. For example, when the resolution is half in the lateral andvertical direction, the size of the square is doubled in lateral andvertical direction.

In FIG. 18, the pixels of the super-resolution image data arerepresented by hollow circles, and sample points corresponding to thelow-resolution image data of half in resolution are represented by solidcircles.

When the pixel values of the initially estimated super-resolution imageare applied to the pixels of the super-resolution image data, the samplevalue of the initially estimated super-resolution image at a samplepoint 1704 is calculated as an average value of the pixel values ofpixels from 1705 to 1708. This is a case in which the sample point 1704is located at the center of the pixels of the super-resolution imagedata therearound.

When the position is displaced as a sample point 1709, the weightedaverage of the portion overlapped by a square 1710 which is representedby the sample point is determined as the sample value of the initiallyestimated super-resolution image. For example, the weight with respectto a pixel 1711 is determined by converting the surface area of ahatched portion 1712 into a weight. Nine squares overlapped with thesquare 1710 are weight so as to be proportional to the overlappedsurface area, and then the weighted average is obtained from the ninepixel values as a sample value of the initially estimatedsuper-resolution image.

If the super-resolution image data is accurate, the sample value imagedas the low-resolution image data should match the sample value of theinitially estimated super-resolution image.

However, they do not match normally. Therefore, the pixel value of theinitially estimated super-resolution image is renewed so as to match.The difference between the sample value and the preliminary sample valueis obtained, and the difference is added to or subtracted from the pixelvalue of the initially estimated super-resolution image to eliminate thedifference. Since there is a plurality of pixel values, the differenceis divided by the weight used in sampling, and is added to or subtractedfrom each pixel value. Accordingly, the sample value and the samplevalue of the initially estimated super-resolution image matches asregards the sample point calculated at this time. In the renewalprocessing on another sample point, however, the pixel data of the samesuper-resolution image may be renewed. Therefore, this renewal processis repeated several times for every sample point. Since thesuper-resolution image data becomes closer to the accurate one graduallyby this repetition, the image obtained after the repetition by thepredetermined number of times is outputted as the super-resolution imagedata.

In this manner, one of the methods of obtaining the pixel value of thesuper-resolution image data by solving the conditional expression withthe pixel value of the super-resolution image data used as an unknownvalue, which gives a condition that the sample value of the estimatedsuper-resolution image obtained from the unknown value to be equal tothe sample value by the pixel value of the actually imagedlow-resolution image data is the POCS method, and IterativeBack-Projection method (for example, see S. Park, et. al,“Super-Resolution Image Reconstruction: A Technical Overview” p. 31) orMAP method, (for example, see S. Park, et. al, “Super-Resolution ImageReconstruction: A Technical Overview” p. 28) may be used as alternativemethods for solving these conditional expressions.

FIG. 19 is a flowchart for establishing the conditional expressions forthe super-resolution.

As shown in FIG. 19, in Step S1801, the above-described conditionalexpressions are established for the pixels of the low-resolution imagedata, that is, for the respective sample values.

Subsequently, in the Step S1802, the pixel value of the super-resolutionimage data is obtained by solving the conditional expressions as thesimultaneous equations.

FIG. 20 shows a state of the brightness value of a certain line insuper-resolution images obtained by applying the cubic convolutionmethod in the related art and the method according to this embodiment toa certain still image. The lateral axis represents the pixel, and thevertical axis represents the brightness value. This is an enlargement ofa portion of a white line in a large light-and-shade image.

As will be seen in FIG. 20, the darkness is emphasized in the darkportion indicated by 633 in the y-coordinate, and the brightness isemphasized in the bright portion indicated by 637 in the y-coordinate.

1. A super-resolution device, comprising: a candidate area setting unitconfigured to set at least one of a plurality of pixels included inimage data as a target pixel, the image data including the plurality ofpixels arranged in a screen and corresponding pixel values representingpixel brightness of the pixels, an area including the target pixel andpixels in the periphery of the target pixel as a target pixel areawithin the screen, and a search area for searching a plurality of changepatterns of pixel values of the pixels included in the target pixel areawithin the screen; an over sampling unit configured to interpolateanother pixel between pixels of the image data; a matching differencecalculating unit configured to calculate a difference between a firstchange pattern of the pixel values of the pixels included in the targetpixel area and a second change pattern of pixel values of pixelsincluded in the search area, the pixels included in the search areaincluding a searched pixel in the search area and the pixels in theperiphery of the searched pixels; an integral-accuracy-vectorcalculating unit configured to compare differences of change patterns ofrespective pixels in the search area calculated by the matchingdifference calculating unit to obtain a pixel position with minimumdifference and calculate an integral-accuracy-vector starting from thetarget pixel and terminating at the search pixel; an extrapolated vectorcalculating unit configured to calculate an extrapolated vector of theintegral-accuracy-vector from the search pixel to a pixel on the screenwhich is not included in the search area using theintegral-accuracy-vector; and a super-resolution pixel value calculatingunit configured to calculate a pixel value of a super-resolution imagehaving a number of pixels larger than a number of pixels included in theimage data on the basis of the integral-accuracy-vector, theextrapolated vector, and pixel values obtained from the image data.
 2. Asuper-resolution device, comprising: a candidate area setting unitconfigured to set at least one of a plurality of pixels included inimage data as a target pixel, the image data including the plurality ofpixels arranged in a screen and corresponding pixel values representingpixel brightness of the pixels, an area including the target pixel andpixels in the periphery of the target pixel as a target pixel areawithin the screen, and first and second search areas for searching aplurality of change patterns of the pixel values of the pixels includedin the target pixel area within the screen; an over sampling unitconfigured to interpolate another pixel between pixels of the imagedata; a matching difference calculating unit configured to calculatefirst and second differences between change patterns of the pixel valuesof the pixels included in the target pixel area and first and secondchange patterns of the pixel values of the pixels included in the firstand second areas, the pixels included in the first and second areasincluding first and second searched pixels in the first and secondsearch areas and the pixels in the periphery of the first and secondsearched pixels in the interpolated image data; anintegral-accuracy-vector calculating unit configured to comparerespective differences of change patterns of the first and second searcharea calculated by the matching difference calculating unit to obtain afirst pixel position and a second pixel position with a minimumdifference respectively, and calculate first and secondintegral-accuracy-vectors starting from the target pixel and terminatingrespectively at the first and second pixel positions; an interpolatedvector calculating unit configured to calculate interpolated vectors ofthe first and second integral-accuracy-vectors from the first and secondpixel positions to a pixel on the screen which is not included in thefirst and second search areas using the first and secondintegral-accuracy-vectors; and a super-resolution pixel valuecalculating unit configured to calculate a pixel value of asuper-resolution image having a number of pixels larger than the numberof pixels included in the image data on the basis of theintegral-accuracy-vectors, the interpolated vector and pixel valuesobtained from the image data.
 3. A super-resolution device comprising: acandidate area setting unit configured to set at least one of aplurality of pixels included in image data as a target pixel, the imagedata including the plurality of pixels arranged in a screen andcorresponding pixel values representing pixel brightness of the pixels,an area including the target pixel and pixels in the periphery of thetarget pixel as a target pixel area within the screen, and a search areafor searching a plurality of change patterns of the pixel values of thepixels included in the target pixel area within the screen; an oversampling unit configured to interpolate another pixel between pixels ofthe image data; a matching difference calculating unit configured tocalculate a difference between a first change pattern of the pixelvalues of the pixels included in the target pixel area and a secondchange pattern of pixel values of pixels included in the search area,the pixels in the search area including a searched pixel in a searcharea and the pixels in the periphery of the searched pixel; anintegral-accuracy-vector calculating unit configured to comparedifferences of the change patterns first and second by the matchingdifference calculating unit to obtain a first pixel position with aminimum difference and calculate an integral-accuracy-vector startingfrom the target pixel and terminating at the search pixel; a congruentvector calculating unit configured to calculate a congruent vector ofthe integral-accuracy-vector from the search pixel to a pixel on thescreen which is not included in the search area using theintegral-accuracy-vector; and a super-resolution pixel value calculatingunit configured to calculate a pixel value of a super-resolution imagehaving a number of pixels larger than a number of pixels included in theimage data on the basis of the integral-accuracy-vector, the congruentvector, and pixel values obtained from the image data.
 4. Asuper-resolution method, comprising: setting at least one of a pluralityof pixels included in an image data as a target pixel, the image dataincluding the plurality of pixels arranged in a screen and correspondingpixel values representing pixel brightness, setting an area includingthe target pixel and pixels in the periphery of the target pixel as atarget pixel area, setting a search area for searching a plurality ofchange patterns of the pixel values of the pixels included in the targetpixel area within the screen; interpolating another pixel between thepixels of the image data in which the target pixel area and the searcharea are set to generate an interpolated image data; calculating adifference between the change pattern of the pixel values of the pixelsincluded in the target pixel area and a change pattern of the pixelvalues of the pixels included in the search area, the pixels in thesearch area including a searched pixel in the search area and the pixelsin the periphery of the searched pixels; comparing differences of thechange patterns of the respective pixels in the search area to obtain afirst pixel position with the minimum difference and calculating anintegral-accuracy-vector starting from the target pixel and terminatingat the search pixel; calculating an extrapolated vector of theintegral-accuracy-vector from the search pixel to a pixel on the screenwhich is not included in the search area using theintegral-accuracy-vector; and calculating a pixel value of asuper-resolution image having a number of pixels larger than a number ofpixels included in the image data on the basis of theintegral-accuracy-vector, the extrapolated vector, and pixel valuesobtained from the image data.
 5. A super-resolution method, comprising:setting at least one of a plurality of pixels included in an image dataas a target pixel, the image data including the plurality of pixelsarranged in a screen and corresponding pixel values representing pixelbrightness of the pixels, setting an area including the target pixel andpixels in the periphery of the target pixel a as target pixel area,setting first and second search areas for searching a plurality ofchange patterns of the pixel values of the pixels included in the targetpixel area within the screen; interpolating another pixel between thepixels of the image data in which the target pixel area and the firstand second search areas are set to generate an interpolated image data;calculating first and second differences between the change pattern ofthe pixel values of the pixels included in the target pixel area andfirst and second change patterns of the pixel values of the pixelsincluded in the first and second areas, said pixels included in thefirst and second areas including the first and second searched pixels inthe first and second search areas and the pixels in the periphery of thefirst and second searched pixels in the interpolated image data;comparing the respective differences of the change patterns of the firstand second search areas to obtain a first pixel position and a secondpixel position with a minimum difference respectively, and calculatingfirst and second integral-accuracy-vectors starting at the target pixeland terminating respectively at the first and second pixels; calculatingthe interpolated vectors of the first and secondintegral-accuracy-vectors from the first and second pixels to a pixel onthe screen which is not included in the first and second search areasusing the first and second integral-accuracy-vectors; and calculating apixel value of a super-resolution image having a number of pixels largerthan the number of pixels included in the image data on the basis of theintegral-accuracy-vectors, the interpolated vector and a pixel valuesobtained from the image data.
 6. A super-resolution method, comprising:setting at least one of a plurality of pixels included in image data asa target pixel, the image data including the plurality of pixelsarranged in a screen and corresponding pixel values representing thebrightness, setting an area including the target pixel and pixels in theperiphery of the target pixel as a target pixel area, setting a searcharea for searching a plurality of change patterns of the pixel values ofthe pixels included in the target pixel area within the screen;interpolating another pixel between the pixels of the image data inwhich the target pixel area and the search area are set to generate aninterpolated image data; calculating a difference between a changepattern of pixel values of the pixels included in the target pixel areaand a change pattern of pixel values of the pixels included in thesearch area, the pixels in the search area including the searched pixelin a search area and the pixels in the periphery of the searched pixel;comparing differences of the change patterns of the respective pixels inthe search area to obtain a first pixel position with the minimumdifference and calculating an integral-accuracy-vector starting from thetarget pixel and terminating at the search pixel; calculating acongruent vector of the integral-accuracy-vector from the search pixelto a pixel on the screen which is not included in the search area usingthe integral-accuracy-vector; and calculating a pixel value of asuper-resolution image having a number of pixels larger than a number ofpixels included in the image data on the basis of theintegral-accuracy-vector, the congruent vector, and a pixel valuesobtained from the image data.